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1.0  INTRODUCTION 


Cockpit  display  systems  are  used  today  to  provide  pilots  and  sensor 
operators  with  radar,  infrared,  or  television  information  which  is  used  for 
navigation,  intelligence  gathering,  target  detection,  recognition,  identifica- 
tion, and  designation.  In  many  cases,  these  tasks  require  high  quality  imagery 
for  optimum  operator  performance.  Unfortunately,  limitations  in  the  display 
or  sensor  system,  or  both,  may  lead  to  degraded  performance  in  meeting 
mission  objectives.  Techniques  for  real  time  enhancement  of  sensor  imagery 
show  promise  for  improving  operator  performance  with  a minimal  increase 
in  system  complexity. 

1.  1 BACKGROUND 

Perhaps  the  most  well-known  and  spectacular  examples  of  image 
enhancement  are  those  employed  to  improve  pictures  transmitted  from  the 
moon  and  planets  as  part  of  our  space  program.  In  these  cases,  large  com- 
puters and  long  processing  times  are  used  to  achieve  dramatic  results  by 
correcting  for  geometric  distortion,  eliminating  noise,  correcting  for  video 
non-uniformity  across  the  image,  and  improving  image  contrast. 

Desirable  as  these  processes  may  be,  their  implementation  is  not 
feasible  in  a modern  military  aircraft.  The  cockpit  environment  is  one  in 
which  processing  delays  of  more  than  a few  milliseconds  are  not  acceptable 
(this  is  what  is  meant  by  real-time  processing)  and  weight,  size,  power, 
and  cost  must  be  minimized. 

This  program  has  been  directed  at  developing  techniques  for  enhancing 
sensor  imagery  to  improve  the  performance  of  the  human  operator.  A major 
program  objective  is  the  enhancement  of  sensor  images  by  adjusting  the 
image  data  to  (1)  better  fit  within  the  performance  constraints  of  the  display 
device  or  (2)  to  overcome  limitations  in  sensor  performance. 
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An  example  of  the  former  is  infrared  (IR)  imagery  which  has  a much 
larger  dynamic  range  than  the  cathode -ray  tubes  on  which  it  is  displayed. 

If  an  attempt  is  made  to  maintain  the  dynamic  range  within  the  limits  of  the 
CRT,  much  information  may  be  lost  to  the  operator  because  of  saturation  or 
compression  of  the  video. 

In  the  second  case,  the  relatively  slow  response  of  a FLIR  sensor 
along  the  scan  lines  tends  to  blur  edges  in  the  picture,  making  target 
recognition  and  identification  more  difficult. 

1.2  OBJECTIVES 

The  major  objective  of  this  program  was  to  evaluate  three  image 
enhancement  techniques  capable  of  improving  operator  performance  utilizing 
sensor  display  systems.  The  techniques  under  evaluation  were  all  judged 
suitable  for  implementation  and  utilization  in  the  cockpit  environment. 
Evaluation  was  based  on  subjective  analysis  of  picture  quality  improvement 
after  the  enhancement  technique  had  been  applied  and  on  physical  data  such 
as  complexity  and  cost.  The  three  techniques  that  were  evaluated  are: 

Local  Area  Brightness  and  Gain  Control  (LABGC),  Local  Area  Histogram 
Equalization  (LAHE),  and  Haar  Transform  Crispening  (HTC). 

1.3  METHODOLOGY 

In  the  performance  of  this  program,  four  major  tasks  were  performed: 

1.  Selection,  collection,  and  digitizing  of  representative  sensor 
data. 

2.  Development  and  optimization  of  the  respective  enhancement 
techniques. 

3.  Processing  of  the  selected  images  by  each  of  the  enhancement 
processes,  and 

4.  Evaluation  of  the  resulting  images. 

Representative  samples  of  imagery  from  each  of  three  major  sensor  types 
(IR,  TV,  Radar)  were  selected.  A flying  spot  scanner  was  used  tc  digitize 
the  input  image  data  from  photographic  transparencies.  A digital  scan 
converter  was  used  to  capture  and  digitize  frames  of  raw  FLIR  sensor  data 
from  magnetic  video  tape.  Radar  image  data  was  collected  directly  from 
computer -compatible  digital  tapes. 
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Software  for  the  three  enhancement  techniques  (LABGC,  LAHE..  and 
HTC)  was  developed  on  the  Hughes  Real  Time  System  Simulation  Facility 
Sigma  5 digital  computer.  Each  technique  was  refined  to  produce  the  best 
enhancement  possible  within  real-time  cockpit  environment  constraints. 
Enhancement  parameters  such  as  gain,  window  size,  and  filter  shapes  were 
empirically  adjusted  until  the  resultant  images  showed  the  most  improvement 
possible  in  the  opinion  of  the  experimenters. 

After  the  images  selected  for  use  in  this  study  were  processed,  the 
resulting  enhancements  were  photographically  recorded.  The  photographs 
(including  those  of  the  unprocessed  image)  were  used  during  the  subjective 
evaluation  to  determine  relative  merits  of  each  processing  technique. 

Subjective  evaluation  of  the  processing  techniques  was  accomplished 
by  asking  each  of  ten  experienced  subjects  to  rate  (on  a scale  of  1 to  7 with 
a rating  of  4 defined  as  "normal")  the  quality  of  each  of  the  processed  images. 
The  rating  was  applied  to  a spectrum  of  operator  tasks  ranging  from  vehicle 
and  sensor  control  through  target  detection,  recognition,  and  identification. 
Thus  the  effectiveness  of  an  enhancement  technique  could  be  measured  as  a 
function  of  sensor  and  task.  Further  evaluation  consisted  of  designing  a 
tentative  hardware  mechanization  of  each  technique  for  estimating 
complexity. 

1.4  SUMMARY 

Several  conclusions  were  drawn  from  this  study.  All  techniques 
produced  the  expected  observable  results  in  the  sensor  imagery.  Contrast 
was  enhanced  and  edges  in  FLIR  imagery  were  indeed  sharpened. 

The  effects  of  the  enhancement  processes  on  improved  operator  per- 
formance were  most  notable  with  the  contrast  enhancement  processes  applied 
to  FLIR  and  TV  imagery.  Although  noticeable  changes  were  produced  in 
radar  imagery  by  all  enhancement  techniques,  the  subjective  evaluations  did 
not  indicate  that  significant  improvements  in  operator  performance  would  be 
achieved.  The  evaluation  data  is  inconclusive  regarding  the  effects  of  these 
enhancement  techniques  (LAHE,  LABGC,  HTC)  on  radar  imagery.  Further 
study  in  this  area  is  warranted. 

It  is  estimated  that  LABGC  and  HTC  would  be  much  (by  a factor  of 
about  2:1)  less  complex  to  implement  than  LAHE. 
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In  summary,  one  can  conclude  from  this  study  that  of  the  three 
enhancement  techniques  investigated,  local  area  brightness  and  gain  control 
offers  the  potential  for  significant  improvement  in  performance  when  applied 
to  electro -optical  sensor  imagery  particularly  at  mid-  and  close-  *anges. 
Estimates  of  hardware  complexity  suggest  that  implementation  of  this 
technique  for  use  with  flight  hardware  is  feasible  and  well  within  the  state 
of  the  art.  Further  evaluation  of  this  technique  through  real-time  man-in-the- 
loop  simulation  or  flight  test  will  yield  more  quantitative  data  on  the  cost 
effectiveness  of  LABGC  in  the  mission  environment. 

Subsequent  sections  of  this  report  contain  descriptions  of  the  image 
collection  processes  (Section  2.  0),  detailed  discussions  of  the  enhancement 
techniques  (Section  3.0),  a presentation  of  the  evaluation  results  (Section  4.0), 
and  conclusions  of  the  study  (Section  5.0). 
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2.0  IMAGE  COLLECTION 


j The  images  used  in  evaluating  the  various  processing  techniques 

were  digitized  from  several  sources:  radar,  FLIR  and  television.  In  all 
i cases,  the  imagery  was  reduced  to  a digital  format  consisting  of  256  lines 

| and  256  elements  per  line.  Each  picture  element,  or  pixel,  was  quantized 

t to  6 binary  bits  representing  64  discrete  gray  scale  levels.  This  resolution 

and  brightness  quantization  are  sufficient  to  produce  images  of  adequate 
quality  for  evaluation. 

The  following  discusses  three  devices  that  were  used  to  digitize 
images  for  this  study: 

1.  Flying  spot  scanner  (FSS) 

2.  Digital  scan  converter  (DSC) 

3.  Radar  direct  recording 

The  FSS  device  allows  the  dig',  ization  of  photographic  transparencies. 
It  consists  of  a cathode  ray  tube  that  g nerates  a bright  spot  whose  position 
is  under  computer  control.  This  spot  is  focused  onto  the  image  transparency 
by  a lens.  The  amount  of  light  passing  through  the  transparency  at  that 
point  is  collected  by  another  lens  and  focused  onto  a photomultiplier  tube 
(PMT).  The  PMT  output  is  converted  from  analog  to  digital  form  to 
produce  a digital  brightness  value  as  shown  in  Figure  2-1.  This  process 
is  repeated  until  the  spot  has  scanned  the  entire  transparency.  The  digital 
numbers  collected  for  each  spot  on  the  transparency  then  make  up  the 
digital  image.  Images  are  then  stored  on  disk  or  tape  for  future  processing. 
Images  collected  this  way  are  shown  in  Figures  4-la,  4-2a  and  4-3a. 
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Figure  2-1.  Flying  spot  scanner  digitizing  process. 


The  second  method  of  image  digitizing  is  through  the  use  of  a digital 
scan  converter  (DSC).  A DSC  is  primarily  used  to  buffer  information  so  that 
sensor  video  (e.g,  radar  video),  entering  at  one  rate  and  format,  can  be  out- 
put at  another.  The  buffer  is  a digital  memory  capable  of  storing  one  picture 
frame.  The  frame  can  enter  the  memory  at  real-time  TV  rates  and  be  read 
out  at  much  slower  computer  rates.  Figure  2-2  illustrates  the  functional 
operation  of  the  DSC  as  it  was  used  to  digitize  FUR  images  from  video  tape. 

The  DSC  was  used  to  capture  single  frames  from  a TV  compatible 
Forward  Looking  Infrared  (FLIR)  sensor  system.  This  was  done  by  monitoring 
the  FLIP,  imagery  as  it  was  played  on  the  video  tape  recorder.  Depressing  a 
switch  at  the  monitor  caused  the  desired  image  to  be  captured  in  the  DSC 
memory  for  subsequent  read-out  by  the  computer.  The  computer  then  trans- 
ferred the  digital  image  from  the  DSC  memory  to  magnetic  tape  or  disc.  FLIR 
scenes  captured  this  way  are  shown  in  Figures  4-4a,  4-5a,  4 -6a,  and  4-7a, 

This  technique  was  used  because  it  overcomes  some  of  the  problems 
of  digitizing  photographs  of  FLIR  displays  using  the  FSS.  When  using  film 
and  a cathode  ray  tube,  information  and  dynamic  range  is  lost  due  to  limita- 
tions in  the  brightness  transfer  function  or  gamma  of  both  the  CRT  and  film. 
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Figure  2-2.  Functional  operation  of  the  digital  scan  converter  (DSC) 


Capturing  FUR  images  directly  from  video  tape  preserves  the  inherent 
characteristics  of  FUR  imagery.  The  four  pictures  collected  and  digitized 
for  this  study  are  typical  of  FUR  imagery. 

The  radar  images  collected  for  this  study  were  directly  recorded  on 
digital  magnetic  tape  during  flight  test  of  the  Forward  Looking  Advanced 
Multi-mode  Radar  (FL-AMR).  These  tapes  are  initially  wide  band  recordings 
that  are  converted  to  computer  compatible  tapes  (CCT)  on  the  ground.  The 
images  thus  recorded  are  from  the  radar's  doppler  beam  sharpened  mode 
(DBS)  which  is  similar  to  a synthetic  array  type  of  ground  mapping  radar. 
Four  scenes  were  selected  that  are  representative  of  radar  imagery  and 
are  shown  in  Figures  4 -8a,  4 -9a,  4-1  Oa,  and  4-1  la. 
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3.0  ENHANCEMENT  TECHNIQUES  AND  RESULTS 


Three  image  enhancement  techniques  are  described  in  this  section: 
two  contrast  enhancement  techniques  and  one  resolution  enhancement  techniqu 
Local  Area  Brightness  and  Gain  Control  (LABGC)  and  Local  Area  Histogram 
Equalization  (LAHE)  comprise  the  contrast  enhancement  techniques.  The 
resolution  enhancement  is  achieved  through  Haar  Transform  Crispening 
(HTC).  Each  of  these  will  be  described  in  detail. 

3.  1 LOCAL  AREA  BRIGHTNESS  AND  GAIN  CONTROL 

This  section  describes  a contrast  enhancement  technique  that  utilizes 
two  prominent  variables  of  display  systems:  brightness  and  gain.  These 
variables  help  match  the  image  video  offset  and  dynamic  range  to  the  display 
such  that  the  operator  can  extract  the  most  useful  visual  information.  How- 
ever, these  variables  are  constant  over  an  entire  picture  frame.  While  for 
the  most  part  this  is  acceptable,  no  real  enhancement  can  be  achieved  by 
their  manipulation.  An  increase  in  video  gain  does  increase  overall  image 
contrast  but  detail  in  dark  and  light  portions  saturate  due  to  the  limited 
dynamic  range  of  the  display  device  (usually  a CRT).  What  is  needed  is  a 
way  of  increasing  the  contrast  in  small  areas  of  an  image  without  causing 
saturation  in  other  areas.  Continuous  adjustment  of  gain  and  brightness 
in  small  areas  throughout  the  picture  would  achieve  this  purpose. 

Local  area  brightness  and  gain  control  (LABGC)  was  achieved  by 
sliding  a small  window  through  the  image  and  adjusting  the  center  element 
by  gain  and  brightness.  Figure  3-1  depicts  this  window  which  was  chosen 
to  be  9 pixels  by  9 lines.  Earlier  attempts  at  LABGC  consisted  of  first 
adjusting  the  brightness  of  the  window's  center  element  and  then  applying  a 
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Figure  3-1.  Sliding 
window  for  local  area 
brightness  and  gain 
control. 

constant  gain  factor.  This  process  can  be  best  illustrated  by  the 
one-dimensional  example  of  Figure  3-2.  Figure  3-2a  shows  a typical  line  of 
video  whore  small  detail  is  present  in  both  the  dark  and  light  regions.  This 
example  shows  6 bit  digital  video  where  0 is  the  maximum  black  and  63  is  the 
maximum  white.  The  first  step  of  the  LABGC  process,  shown  in  Figure  3-2b, 
is  to  adjust  the  brightness  toward  mid-gray,  which,  digitally,  is  32.  This  is 
done  by  adding  to  the  center  pixel  the  difference  between  mid-gray  (32)  and 
the  average  brightness  of  9 pixels.  Notice  that  this  produces  a video  which 
tends  to  be  flat  and  washed  out.  However,  this  step  is  necessary  in  order  that 
the  applied  gain  function  does  not  cause  saturation.  The  next  step,  shown  in 
Figure  3 -2c  is  the  application  of  a gain  function  to  the  brightness  adjusted 
video  of  Figure  3-2b.  This  is  done  by  multiplying  the  video  by  a constant  gain 
uniformly  across  the  line.  The  two-dimensional  visual  counterpart  of  this 
process  is  shown  by  the  tank  image  of  Figure  3-3,  where  a,  b,  and  c corre- 
spond to  a,  b,  arid  c of  Figure  3-2.  Notice  that  the  processed  image  tends  to 
take  on  a binary  appearance  - there's  little  mid  gray  information.  A consid- 
erable amount  of  detail  has  been  brought  out  in  the  image  but  it  is  not 
pleasing.  The  obvious  way  to  avoid  the  binary  problem  is  to  decrease  the 
gain  factor.  Figure  3-4  shows  the  effect  of  lower  uniform  gain.  This  helps, 
but  it  appears  desirable  to  maintain  a high  gain  in  some  areas  of  the  image 
while  reducing  it  in  others.  For  example,  a lower  gain  in  the  grass  area 
would  help  maintain  a grass-like  texture  but  a higher  gain  in  the  tread  area 
would  help  to  enhance  it. 
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c.  GAIN  APPLIED 


Figure  3-2.  One -dimensional  example  of 
local  brightness  and  gain  control. 


3-3 


a.  Original  256  x 256  resolution 
x 6 bit  video. 


b.  Brightness  adjusted. 


c.  Gain  = 10  applied. 


Figure  3-3. 


Example  of  local  area  brightness  and  gain  control 
with  uniform  gain. 
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c)  gain  = 4 


d)  gain  = 5 


Figure  3-4.  Effects  of  differing  levels  of  uniform  gain 
with  brightness  adjusted  toward  mid-gray. 
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This  latter  effect  can  be  achieved  by  making  the  brightness  and  gain 
vary  as  a function  of  the  local  statistics  within  the  image.  If  the  gain  is  a 
function  of  the  standard  deviation  of  pixel  values  within  the  sliding  window, 
then  the  gain  can  be  made  larger  in  areas  with  little  gray  scale  level  change 
and  smaller  in  areas  of  large  change.  Figure  3-5  illustrates  in  histogram 
form  the  statistics  of  different  local  areas  of  the  image.  If  the  gain  function 
applied  to  the  image  is  inversely  proportional  to  the  standard  deviation  as  in 
Figure  3-6  the  desired  effect  is  achieved.  One  way  to  apply  this  gain  is 
about  the  local  statistical  mean  of  the  video.  But  this  causes  a problem; 
certain  pixels  may  exceed  the  bounds  of  maximum  black  or  maximum  white. 
This  problem  can  be  overcome  by  applying  a brightness  control  to  bring  the 
pixel  intensities  back  within  bounds.  Figure  3- 7b  illustrates  the  application 
of  video  gain  about  the  local  mean  to  improve  contrast.  Figure  3 -7c  shows 
the  application  of  the  brightness  control  to  avoid  saturation.  The  combina- 
tion of  a gain  function  based  on  the  local  standard  deviation  and  a brightness 
bias  control  comprise  the  LABGC  contrast  enhancement  technique. 
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Figure  3-5.  Local  image  statistics. 
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STANDARD  DEVIATION 

Figure  3-6.  Gain  as  a function  of  standard  deviation 
within  sliding  window. 


Figure  3-7.  Statistical  gain  and  brightness  control. 


In  more  mathematical  terms,  the  LABGC  technique  is  described  as 
follows.  Within  the  9x9  sliding  window  the  mean  and  standard  deviation 
are  computed  as 


MEAN  = M 
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and 


STANDARD  DEVIATION 


where 

I (i,j)  = a pixel  vithin  the  sliding  window. 

Then  the  gain  function  as  shown  in  Figure  3-6  is  given  by 


GAIN  = G = A - KS 

where  A is  the  upper  limit  of  the  gain,  K is  the  slope  of  the  function,  and 
S is  the  standard  deviation  within  the  window.  If  G < 1,  then  G is  set  to  1. 

The  computation  of  the  brightness  control  value  is  based  on  the 
following  logic.  The  majority  of  the  pixels  in  the  sliding  window  lie  within 
the  standard  deviation  about  the  mean.  Hence,  most  intensities  are  bounded 
by  M - S and  M + S.  This  information  can  be  used  to  determine  if  the  gain 
adjusted  video  will  exceed  the  black  or  white  intensity  boundaries.  Fig- 
ure 3 -7b  depicts  a situation  where  maximum  black  is  exceeded  after  the 
gain  is  applied.  The  value  of  the  brightness  bias  control  for  this  case  is  the 
distanc2  that  the  video  distribution  extends  beyond  0 or  maximum  black. 

Thus  the  brightness  bias  is  GS  - M.  Similarly,  for  the  maximum  white 
boundary  (63)  the  brightness  bias  is  given  by  63  - (M  + GS). 


This  brightness  control  logic  works  well  for  the  majority  of  video 
distributions.  However,  certain  distributions  arise  where  the  above  logic 
produces  picture  anomalies.  Figure  3-8b  has  been  processed  by  the  LABGC 
technique  using  the  above  brightness  control  logic.  Note  the  "bordering"  that 
occurs  at  high  contrast  edges.  In  particular,  notice  the  white  dot  in  the  black 
bay  area  at  the  center  left  of  the  original  image  (3- 8a).  In  the  enhancement 
(3-8b)  a 9 x 9 window  anomaly  has  been  produced  at  this  point.  Figure  3-9a 
illustrates  in  histogram  form  the  statistics  of  this  local  area  in  the  original 
image.  When  the  gain  and  brightness  bias  control  is  applied  lo  the  image  as 
in  Figure  3-8c  the  resulting  distribution  is  biased  more  than  necessary  to  pre- 
vent the  enhanced  pixels  from  exceeding  maximum  black  as  they  do  in  3-9b. 
This  problem  is  solved  by  limiting  the  brightness  bias  value  when  the  standard 
deviation  does  not  accurately  represent  the  video  distribution  which  in  this 
case  is  to  the  left  of  the  mean.  A better  brightness  bias  value  is  MG  - M 
which  can  be  seen  in  Figure  3-9b.  For  the  maximum  white  limit  the  bias 
value  using  this  logic  is 


B = 63  - (M  + G (63-M)). 

If  the  brightness  bias  value  (B)  is  computed  using  a combination  of 
both  the  mean  and  standard  deviation  logic  discussed  above,  then  the  center 
element  of  the  sliding  window'  is  enhanced  as  follows, 

I (5,  5)  = [i  (5,  5)  - m)  *G  + B + M 

A 

where  I (5,5)  is  the  enhanced  center  window  element.  Figure  3-8c  shows 
the  results  of  this  combined  brightness  control  logic  and  a statistical  gain. 

3.  2 LOCAL  AREA  HISTOGRAM  EQUALIZATION 

Histogram  equalization  is  a unique  image  contrast  enhancement 
technique.  Historically,  it  has  been  applied  only  to  an  entire  picture  frame. 
This  technique  is  called  full-frame  histogram  equalization  (FFHE).  What  is 
described  here  is  a procedure  for  applying  histogram  equalization  to  a 
smaller  portion  or  local  area  of  a picture  frame.  The  advantage  of  the 
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a)  Original 


b)  LABGC  with  standard 
deviation  brightness 
control  only 


c)  LABGC  with  both  standard 
deviation  and  mean 
brightness  control 


Figure  3-8.  Example  of  "bordering”  due  to  improper 
brightness  control. 
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Figure  3-9.  Distributions  causing  "bordering"  due  to 
improper  brightness  control. 

local  area  histogram  equalization  (LAHE)  technique  is  that  it  eliminates 
contrast  attenuation  in  image  areas  that  represent  a statistically  small  but 
important  portion  of  the  gray  scale  histogram.  The  following  paragraphs 
review  FFHE  and  then  describe  the  LAHE  technique. 

An  image  histogram  is  formed  by  accumulating  and  tabulating  the 
occurrences  of  each  pixel  (picture  element)  intensity.  For  the  example  in 
Figure  3-10,  the  intensities  range  over  64  quantized  levels.  The  histogram 
for  the  entire  image  is  characterized  in  Figure  3-11.  This  distribution 
indicates  that  the  picture  is  generally  light  with  few  areas  in  the  mid-gray 
range  and  some  predominantly  very  dark  areas.  It  shows  that  the  distribution 
of  gray  scale  information  primarily  lies  at  either  end  of  the  light/dark 
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Figure  3 -10.  56  x 256  x 6 bit 

original  tank  image. 


Figure  3-1 J.  Image  histogram  for 
Figure  3-10. 

intensity  scale.  Thus,  if  the  image  is  displayed  on  a CRT  which  generates 
16  shades  of  gray,  the  result  is  equivalent  to  slicing  the  gray  scale  axis  of 
the  histogram  into  16  equally  spaced  levels  as  shown  ir.  Figure  3-12. 

It  becomes  apparent  that  if  the  amount  of  information  is  related  to 
the  level  of  occurrence  of  each  gray  scale  then  each  intensity  level  in  the 
displayed  image  does  not  carry  the  same  amount  of  information.  If  the 
picture  intensities  are  distributed  as  in  Figure  3-11,  it  might  be  better 
to  use  more  display  gray  scale  levels  where  there  are  more  occurrences  of 
gray  scale  information  in  the  image  and  fewer  levels  where  there  is  less 
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Figure  3-12.  Equal  intensity  level  slicing. 

data.  This  nonuniform  distribution  of  gray  scale  levels  can  be  computed  by 
finding  the  total  area  under  the  histogram  and  dividing  this  area  into  N or, 
for  example,  16  slices  having  equal  areas  as  shown  in  Figure  3-13. 

The  intensity  level  boundaries  for  each  slice  then  define  the  range  of 
original  intensity  levels  that  will  be  assigned  a new  gray  scale  level  for  the 
displayed  image.  When  this  is  done  a histogram  of  the  resulting  displayed 
imago  has  a flat  or  equal  distribution  of  gray  scale  levels  as  in  Figure  3-14. 
Hence  the  name  "Histogram  Equalization.  " 


Figure  3-13.  Equal  area  slicing. 


Figure  3-14.  Histogram  of  histogram  equalized  image. 
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j Figure  3-16  shows  the  effect  of  FFHE.  The  contrast  and  hence  the 

detail  in  the  light  areas  of  the  image,  such  as  the  grass  and  the  top  of  the 
{ tank,  have  been  enhanced.  The  dark  areas  of  the  image,  mainly  around  the 

i tread  area,  have  not  been  enhanced  since  these  brightness  levels  do  not 

show  a high  level  of  occurrence  in  the  histogram.  Figure  3-15  helps  visual- 
ize this  effect.  Note  that  the  area  of  the  histogram  having  low  occurrence 
represents  a part  of  the  image  that  will  have  half  as  many  gray  scale  levels 
in  a processed  picture  than  in  the  original.  Obviously,  if  the  target  of 
interest  lies  in  that  area  of  the  image,  FFHE  processing  would  not  be  an 
enhancement. 

This  difficulty  can  be  resolved  by  using  local  area  histogram  equaliza- 
tion. That  is,  rather  than  redistributing  the  gray  scale  levels  based  on  a 
histogram  representing  the  entire  image,  the  equalization  can  be  carried  out 
on  a two-dimensional  sliding  window  basis.  Thus,  the  intensity  for  any 
particular  point  in  the  image  is  adjusted  according  to  a histogram  of  the  area 
contained  within  a window  immediately  surrounding  the  point  as  shown  in 
Figure  3-17.  In  operation,  this  window  moves  across  the  image  in  two 
dimensions,  horizontally  and  vertically.  These  reassigned  center  picture 
elements  then  make  up  the  processed  image.  On  images  like  the  tank,  detail 
is  greatly  enhanced  in  regions  like  the  tread  rollers  as  shown  in  Figure  3- 16c. 


Figure  3-15.  Potential  deficiency  of  full  frame 
Histogram  equalization. 
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a.  Original 


b)  Full  frame  histogram 
equalized 


c)  Local  area  histogram 
equalized 


Figure  3-16.  Comparison  of  full  frame  and  local  area 
histogram  equalization. 


Figure  3-17.  Two-dimensional 
Histogram  equalization. 
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This  process  of  forming  a histogram  around  each  picture  element  is 
quite  time  consuming.  It  would  be  more  efficient  to  compute  a histogram 
and  equalize  a group  of  picture  elements  rather  than  just  one.  Therefore, 
a trade-off  study  was  performed  to  determine  the  optimum  number  of 
elements  to  equalize  within  each  window  area.  Based  upon  the  histogram  in 
the  sliding  window  of  32  rows  by  32  columns,  the  picture  elements  in  the 
center  N rows  and  N columns  were  adjusted  for  N = 1,  N = 2,  N = 4,  N = 8, 

N = 16,  and  N = 32.  Figure  3-18  illustrates  these  different  size  inner 
windows  starting  in  the  upper  left  hand  corner  of  an  image.  The  large 
outer  box  is  the  32  x 32  window  that  represents  the  boundaries  of  histogram 
formation.  The  heavy  dotted  and  cross-hatched  areas  represent  those 
picture  elements  that  get  equalized.  The  light  dotted  and  lined  areas  repre- 
sent the  next  areas  to  be  equalized.  This  equalization  is  based  on  the  histo- 
gram formed  within  the  boundaries  of  the  Owter  dashed  box.  The  results 
of  these  differing  inner  window  sizes  are  shown  in  Figures  3- 19a  through 
3-J9f.  Each  picture  corresponds  to  the  process  depicted  in  Figures  3-1 8a -f. 

It  can  be  seen  that  for  N = 8,  16,  and  32,  the  enhanced  picture  has  a "boxy" 
look  while  for  N = 1,  2,  and  4,  this  effect  is  absent.  Therefore,  the  conclu- 
sion is  that  the  pixels  in  the  center  4 rows  and  4 columns  of  each  sliding 
window  can  be  equalized  without  undue  picture  distortion  while  producing 
good  computational  speed. 

A potential  problem  arises  in  local  histogram  equalization  when  the 
distribution  is  concentrated  at  only  a few  intensities  as  in  Figure  3-20.  In 
this  particular  example,  the  intensities  run  from  29  to  35.  After  histogram 
equalization,  the  intensity  distribution  would  look  similai  to  that  in 
Figure  3-21.  Notice  that  true  equalization  has  not  been  achieved  since  the 
upper  brightness  levels  are  not  being  utilized.  This  is  due  to  the  fact  that 
there  are  fewer  gray  scale  levels  in  the  area  to  be  enhanced  than  there  are 
display  levels.  When  this  particular  situation  occurs,  the  histogram 
equalization  technique  enhances  the  contrast  but  also  biases  *he  equalized 
video  toward  the  dark  end  of  the  intensity  scale  as  shown  in  Figure  3-21. 

This  problem  is  alleviated  in  the  following  manner.  The  arithmetic  means  of 
intensity  levels  in  the  original  sliding  window  area  and  the  histogram  equalized 
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Figure1  3-18.  Different  inner  window  sizes  for  the  LAHE  technique. 
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Figure  3-20.  Picture  histogram  within  sliding  window  showing 
a small  range  of  intensity  levels. 
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Figure  3-21.  Histogram  of  histogram  equalized  sliding 
window  of  Figure  3-20 
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Figure  3-22. 


Histogram  of  adjusted  histogram  equalized 
sliding  window. 
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data  are  computed.  Based  upon  these  means  the  histogram  equalized 
intensities  are  biased  to  the  right  to  make  the  two  means  correspond.  In  the 
example  above,  the  mean  in  the  original  sliding  window  is  32  and  the  mean 
of  the  intensity  levels  after  histogram  equalization  is  14.  Therefore,  the 
intensity  levels  after  histogram  equalization  are  biased  by  18  so  that  their 
mean  is  now  32  which  corresponds  to  the  original  mean.  The  result  of  this 
biasing  is  shown  in  Figure  3-22.  This  biasing  produces  a contrast  enhanced 
image  whose  mean  brightness  in  local  areas  is  matched  to  the  original. 

In  summary,  the  local  area  Histogram  Equalization  technique  is  a 
powerful  one  for  enhancing  image  contrast.  It  does  so  by  the  non-linear 
assignment  of  gray  scale  levels  in  an  adaptive  manner  based  upon  the  local 
statistics  of  the  image. 

3.3  HAAR  TRANSFORM  FILTERING 

Much  analysis  and  experimental  work  has  been  done  in  the  general 
area  of  spatial  frequency  filtering  of  images.  What  is  discussed  here  is  a 
way  of  achieving  results  similar  to  the  Fourier  Transform  techniques  without 
requiring  an  actual  Fourier  Transform.  The  Haar  transform  is  utilized 
since  it  is  much  faster  and  easier  to  compute.  Table  3-1  lists  various  trans- 
forms that  are  currently  being  used  in  image  processing  work.  The  number 
of  computations  required  for  each  transform  arc  enumerated.  Notice  that 
the  Haar  transform  requires  the  fewest  number  of  computations. 
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TABLE  3 - 1 . IMAGE  TRANSFORMS 


TRANSFORM 

NO. 

OPERATIONS 

N- 

512 

N* 

1024 

TYPE  OPERATIONS 

FOURIER 

2N3 

268.435,456 

2.147.483  648 

COMPLEX  MULTIPLIES  ft  ADDS 

FFT 

2N2LOG2N 

4.718.592 

20,971,520 

COMPLEX  MULTIPLIES  & ADDS 

HADAMARD/WALSH 

2N2LOG2N 

4.718,592 

20.971.520 

ADDS 

HAAR 

523.283 

2.095,104 

ADDS 

CONVOLUTION 

n2xm2 

67.106.884 

1,073,741,824 

MULTIPLIES  AND  ADDS 

(M  • 161 

(M- 

32) 

N 

M 
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CONVOLUTION  FILTER 

M 

3-20 


The  characteristics  of  the  Haar  transform  decomposition  are  not  as 
general  and  useful  as  the  Fourier  and  Hadamard.  However,  the  Haar  decom- 
position is  characterized  by  a differential-like  breakdown  that  suggests  its 
use  for  crispening.  Figure  3-23  shows  three  kernels  or  basis  vectors  for  a 
Fourier,  Hadamard  and  rational  Haar  transform.  Note  that  the  rational  Haar 
waveforms  contain  many  zero  areas  that  result  in  fewer  computations.  The 
Fourier  transform  is  composed  of  both  a sine  and  cosine  function.  Only  the 
sine  waveform  is  shown  in  Figure  3-23.  Note  the  magnitudes  of  the  Fourier 
waveform  are  generally  less  than  1.  This  necessitates  complex  multiplication 
rather  than  simple  addition.  The  Hadamard  and  rational  Haar  waveforms  on 
the  other  hand  have  magnitudes  of  only  1,  -1  and  0.  This  results  in  only 
addition  and  substraction  for  their  computation. 

The  following  section  describes  in  detail  the  derivation  of  the  rational 
Haar  transform.  The  normal  Haar  transform  is  characterized  by  a matrix 
that  contains  entries  that  are  powers  of  ^2~which  eliminates  simple  addition 
and  subtraction.  The  derivation  shows  how  these  y/Zn  terms  can  be  factored 
out  in  such  a way  that  only  addition  and  subtraction  (1,  -1  terms)  and  weight- 
ing factors  are  required  to  perform  the  transform.  For  a combination  of 


forward  and  inverse  Haar  transforms,  the  weighting  factors  are  simple 


Figure  3-23.  8 point  transform  decomposition  waveforms. 
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binary  shifts.  This  is  an  important  result  for  high  speed  computation.  The 
rational  Haar  transform  matrix  can  itself  be  factored  into  a product  of 
matrices  that  each  contain  very  simple  entries.  This  matrix  product  is  the 
basis  for  the  fast  Haar  transform  algorithm  depicted  in  Figure  3-24. 

INPUT  VECTOR  V = (VJ#  V r Vy  Vg,  Vg,  V?,  Vg) 


TRANSFORMED  VECTOR  H V = (Hj,  H2>  Uy  H4>  Hg,  H6>  H?,  Hg) 
Figure  3-24.  Rationalized  Haar  fast  algorithm. 


Derivation  of  Rational  Haar  Transform 


The  following  illustrates  by  example  the  derivation  of  the  rational 

3 

Haar  transform  matrix.  The  case  N = 2 = 8 is  sufficiently  complex  to  form 

a good  example.  The  Haar  transform  matrix  H is 
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, which  can  be 
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factored  as  H 


= F&  where 
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and 
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where  H is  the  rationalized  Haar  matrix.  H itself  can  be  further  factored 
into  a product  of  three  matrices:  Hj,  and 
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Hence,  H = FH3  H2  H1 

The  fact  that  H can  be  factored  into  three  simpler  matrices  gives  rise  to 
a fast  algorithm  for  computing  the  rational  Haar  transform  of  a vector  V. 
This  algorithm  is  depicted  in  Figure  3-24.  Note  that  each  computational 
level  in  the  algorithm  represents  one  matrix  multiply.  To  take  the  Haar 
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transform  of  a vector  V,  the  fast  algorithm  is  first  applied  to  the  vector  and 
then  each  resultant  term  is  weighted  according  to  the  matrix  F.  The  inverse 
Haar  transform  algorithm  is  similar.  For  a combination  of  forward  and 
inverse  Haar  transforms,  the  weighting  terms  reduce  to  powers  of  2,  or 
binary  shifts. 

The  forward  and  inverse  Haar  transform  of  Image  I is 


where 


HTH(I) 


A T A 

(FH)AFHI 


AT  T A 

= HAFAFHI 


a T A 

= H ffhi 


at  2A 
= HAF  HI 


0 0 
1 0 
0 2 
0 0 
0 0 
0 0 
0 0 
0 0 


0 0 0 0 0 

0 0 0 0 0 

0 0 0 0 0 

2 0 0 0 0 

0 4 0 0 0 

0 0 4 0 0 

0 0 0 4 0 

0 0 0 0 4 


The  transpose  of  H,  H , is  employed  in  the  above  equation  as  the  inverse 

-1  T 

Haar  transform  since  H is  real  and  orthogonal,  i.e.,  H = H . In  the 

-1  T 

following  equations  H will  be  used  ir  place  of  H . 


Haar  Transform  Crispening 

High  spatial  frequencies  of  an  image  that  have  been  attenuated  can  be 
restored  via  transform  filtering.  This  is  normally  called  "image  restora- 
tion." "Image  enhancement"  results  when  further  emphasis  or  amplification 
is  placed  or  selected  spatial  frequencies  of  the  image.  Early  attempts  at 
image  "crispening"  or  "edge  enhancement"  were  implemented  by  using  differ- 
ential operators  in  the  spatial  domain,  A generalized  set  of  even  order 
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partial  derivatives  with  constant  coefficients  can  be  used  to  form  the 
crispening  function  as  follows: 

P (x,  y)  = A p (x,  y)  + B 72  p (x,  y)  + C y4  p (x,  y)  + . . . 

where: 

p (x,  y)  = a picture  element  at  location  x,  y 
p (x,  y)  = enhanced  picture  element  at  x,  y 

Vn  = n^  gradient  operator 

6n  + 6n\ 
lx*  Ty^l 

A,  B,  C,  ...  = arbitrary  constants 

The  simplest  crispening  operator  is  given  by 

P (X,  y)  = p(x,  y)-k/P  (X,  y)  (1) 

where  1c  is  a constant. 

The  second  partial  derivative  is  involved  in  the  crispening  operator 

since  it  defines  the  arcs  or  brightness  curvatures  in  the  original  image. 

For  example,  consider  the  one -dimensional  function  p (x)  shown  in  Fig- 

th 

ure  3-25a.  For  one-dimensional  crispening  the  n gradient  operator  can 

th 

be  replaced  with  the  r.  partial  derivative  with  respect  to  the  appropriate 

(,5^  in  \ 

either—  or  -2—1  . The  second  partial  derivative  with  respect 
to  x is  plotted  in  Figure  3-25b.  Figure  3-25c  demonstrates  the  results  of 
applying  Equation  (1)  to  the  function  p (x),  namely,  that  the  function  has  been 
enhanced  by  ’’sharpening"  up  or  "crispening"  the  edges. 
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(b) 


aVax2 


Figure  3-25.  Geometric 
significance  of  2nd 
partial  derivative. 


As  mentioned  before,  the  operator  1-kD  when  applied  to  a function 


p(x)  acts  as  a crispening  operator,  where  D 


2 ~ 62 


We  have  chosen  to 


* 6x2' 

implement  this  one -dimensional  crispening  operator  in  the  Haar  domain  for 
image  enhancement.  The  reason  for  this  is  that  most  resolution  loss  from  a 
sensor  system  is  one -dimensional  in  nature.  Ir  a TV  system,  for  example, 
horizontal  resolution  requires  the  greatest  system  bandwidth.  All  the  Haar 
transform  crispening  done  for  this  study  has  been  done  in  the  horizontal 
dimension. 
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The  following  describes  in  detail  the  derivation  of  the  one -dimensional 
cilspening  function,  G,  that  when  applied  in  the  Haar  domain,  achieves  the 
equivalent  of  the  crispening  operator.  Let  the  following  variables  hold  for 
this  derivation: 

H = Haar  transform 

H * = inverse  Haar  transform 

A 

H = rationalized  Haar  transform 
a-1 

H = inverse  rationalized  Haar  transform 

I = image 

G = Haar  domain  crispening  operator 

A 

I = enhanced  image 

A _ 1 A _1 

As  shown  previously  H = FH  and  H = H F. 

Now  the  spatial  domain  crispening  operator  is 


A 

I = 


I. 


However,  we  wish  to  achieve  similar  crispening  in  the  Haar  domain  such 
that 


I = H"1  [g(HI)]  = (l  - kDx2)  I.  (2) 

From  this  equation  we  can  derive  the  operator  G such  that  when  the  product 

of  G and  the  Haar  transform  of  an  image  is  inverse  Haar  transformed,  the 

2 

equivalent  of  the  spatial  domain  crispening  operator  (1  -kD^  ) results. 

From  (2), 


H_1GHI  = (1  - kD  2)  I 
' x 


and 


H_1GH  = 1 - kD  2 
x 
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then  solving  for  G 


GH  = H - H kDx2 

G = HH-1  - HkD  2 H"1 
x 

= 1 - kHD  2 H”1 
x 

A 

Since  H = FH,  then  substituting 

G = 1 - kFHDx  (FH)  1 
A 2 A -1 

= 1 - kFHDx  H aF  (3) 

Therefore  when  G is  applied  to  the  Haar  transform  of  an  image  via  (2), 
crispening  results.. 

Implementation  of  the  Haar  Crispening  Operator 

The  Haar  transform  domain  crispening  operator  G was  previously 
derived  such  that  an  image  I is  enhanced  by  the  following  equation: 

I = H” 1 GHI  (4) 

where  G = 1 - kFHD  2H_1F. 

x 

Now  by  substitution 

A -1  A 2A-1 
I = H 1 (1  - kFHD  £ H lF)  HI. 
x 

A 

Reducing  the  Haar  transform  to  rational  transforms  where  H = FH,  yields 

A A _ 1 A 7 A 1 A 

I = (FH)  (1  - kFHDx  H lF)  FHI 

A-1  2A  2A-1  ? a 

= (1  - kH  F HD  H F H)  I (5) 
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Letting 


A 


2A  2A-1  > 
F HD  H F ' 
x 


(6) 


we  have  from  (5) 


A A „ I A 

I = ( 1 - kH  AH)  I 


(7) 


where  A now  represents  the  Haar  transform  domain  weighting  function  that 
when  applied  to  the  rationalized  Haar  transform  of  an  image  produces  cris- 
pening.  Note  that  (6)  contains  F matrices  that  are  conveniently  powers  of 
two,  simplifying  hardware  mechanization. 

Since  the  forward  and  inverse  rational  Haar  transforms  are  easy  to 
compute,  the  complexity  of  the  entire  crispening  process  of  equation  (7) 
relies  upon  the  complexity  of  the  matrix  A.  Note  that  the  matrix  A is 
independent  of  the  input  image  and  can,  therefore,  be  computed  before  the 
processing  occurs. 

Consider  the  case  where  the  size  of  the  input  image  is  8 x 8.  Making 

nd  2 

use  of  the  2 order  finite  difference  operator,  can  be  written  in  the 

following  matrix  form: 
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Therefore, 


A = F2HD  2H"1F2  = 4 
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In  practice,  the  implementation  of  the  matrix  A is  prohibitive  due  to 
all  of  its  non-zero  terms.  Therefore,  two  engineering  approximations  can 
be  taken  to  reduce  the  matrix  A to  one  that  can  be  easily  implemented  in 
hardware.  They  ar  to  set  all  off-diagonal  entries  in  A to  zero  and  to  set 
all  diagonal  entries  in  A to  the  nearest  power  of  two.  Mathematically,  the 
two  steps  tend  to  offset  each  other  and,  therefore,  cause  this  to  be  a good 
approximation  to  the  actual  A matrix. 

Therefore,  the  matrix  A approximation  is 


00000000 
0-1  0 0 0 0 0 0 
00  -8  00000 

A_4  000  -8  0000 
0 0 0 0 -320  0 0 
0 0 0 0 0-320  0 
0 0 0 0 0 0 -32  0 
_0  0 0 0 0 0 0 -32_ 

The  crispening  process  using  the  Haar  transform  can  now  be  totally 
implemented  by  using  only  addition,  subtraction  and  binary  shifts.  Fig- 
ure 3-26  summarizes  the  Haar  transform  crispening  technique. 

Now  that  the  derivation  of  the  crispening  operator  is  complete  only 
one  variable  remains  to  be  determined;  the  value  of  k from  equation  (7). 

Note  that  k determines  the  amount  of  crispening  to  be  added  to  the  original 
image.  Figure  3-27  shows  the  effect  of  varying  levels  of  k where  the 
crispening  is  applied  only  to  the  horizontal  dimension.  Inspection  of  the 
results  of  Figure  3-27  shows  that  k = 1 provides  a good  amount  of  crispening 
for  most  cases. 
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e>  k - 4 
Figure  3-27. 


Varying  levels  of  Haar  transform  crispening  du< 
to  value  of  k from  equation  (7). 


3-3; 


?oee*a 


4.  0 EVALUATION  OF  ENHANCEMENT  TECHNIQUES 


This  section  describes  the  results  of  two  evaluations  of  the 
enhancement  techniques  (LAHE,  LABGC,  HTC): 

• Human  Factors  Evaluation 

• Hardware  Implementation  Evaluation 

The  human  factors  evaluation  analyzed  subjectively  the  utility  and 
viewer  acceptance  of  each  enhancement  technique.  This  was  done  by  first 
enhancing  an  ensemble  of  eleven  images  representative  of  3 different  sensor 
systems.  The  best  of  each  technique  was  used.  Then,  ten  subjects  evaluated 
these  enhancements  using  the  original  image  as  a reference. 

The  hardware  implementation  evaluation  was  performed  by  making  a 
functional  block  diagram  of  each  technique.  Based  on  these,  an  estimate  was 
made  of  the  number  of  integrated  circuits  required  for  each. 

The  results  of  these  evaluations  are  described  in  the  following  sections. 

4.  1 IMAGE  ENHANCEMENT  EVALUATION 
Summary 

The  image  enhancement  evaluation  was  conducted  to  obtain  data  on 
viewer  acceptance  and  utility  of  enhanced  images.  Ten  subjects  with  exten- 
sive backgrounds  in  the  development,  display,  and  interpretation  of  sensor 
imagery,  subjectively  evaluated  eleven  tactical  scenes  Figures  4- la  through 
4- 11a  in  terms  of  seven  typical  operational  tasks.  The  evaluation  was  made 
on  a seven  point  scale,  with  the  original  unenhanced  image  presentation  serv- 
ing as  the  base  unit  (4)  and  the  enhanced  images  being  judged  as  better  {5,  6, 
or  7)  or  worse  (3,  2,  or  1)  than  the  original  image. 
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The  resulting  data  support  the  following  conclusions: 

1.  Viewer  acceptance  and  utility  of  enhanced  imagery  is  dependent 
upon  the  sensor  being  used,  the  scene,  and  the  operator  task. 

2.  Enhanced  imagery  is  most  useful  in  detailed  interpretation  tasks 
conducted  at  short  to  medium  range  using  optical  sensors. 

3.  Enhancement  techniques  are  of  unknown  value  when  applied  to 
synthetic  aperture  radar  imagery. 

In  summary,  this  brief  evaluation  strongly  indicates  that  image 
enhancement  techniques  can  be  utilized  to  improve  viewer  interpretation  per- 
formance of  critical  tactical  tasks  which  involve  overall  target  area  orienta- 
tion, target  detection,  identification  and  designation. 

Evaluation  Task 


The  evaluation  task  reflected  the  following  factors: 

1.  Sensors  — Samples  of  TV,  FLIR,  and  synthetic  aperture  radar 
imagery  were  used. 

2.  Target  scenes  — A series  of  11  target  scenes  were  chosen  as 
being  representative  of  a range  of  operational  tactical  target 
scenes. 

3.  Image  enhancement  techniques  — Each  target  scene  was  presented 
in  a set  of  four  photographs  as  follows: 

• Original  unenhanced  sensor  image.  Figures  4- la  through 
4- 11a. 

• A Local  Area  Histogram  Equalized,  4x4  center  box  image. 
Figures  4-lb  through  4-1  lb, 

• A Local  Area  Brightness  and  Gain  Control  statistical  gain 
16/4  image.  Figures  4-lc  through  4-1  lc. 

• A Haar  Transform  Crispened  image  with  k = 1.  Figures 
4- Id  through  4-1  Id. 

4.  Subjects  — 10  subjects  evaluated  the  imagery.  Five  subjects 
were  human  factors  personnel,  five  were  display  systems  engi- 
neers. All  subjects  were  experienced  in  the  interpretation  of 
one  or  more  types  of  sensor  imagery. 

5.  Operational  tasks  — The  imagery  was  evaluated  relative  to  its 
potential  use  in  the  following  set  of  operational  tasks: 

• Vehicle  control 

• Sensor  control 
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a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness  d)  Haar  Transform 

and  Gain  Control,  Crispened,  k = l. 

statistical  gain  16/1. 

Figure  4-1.  TV  imagery  — tank  (close-in). 
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a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness  d)  Haar  Transform 

and  Gain  Control,  Crispened,  k = 1. 

statistical  gain  16/4, 

Figure  4-2.  TV  imagery  — road  (medium  range). 
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a)  Original, 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


r r 
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c)  Local  Area  Brightness  d)  Haar  Transform 

and  Gain  Control,  Crispened,  k=  1. 

statistical  gain  16/4. 

Figure  4-3.  TV  imagery—  terrain  (long  range). 
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a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4 x 4 
center  box. 


c)  Local  Area  Brightness 
and  Gain  Control, 
statistical  gain  16/4, 


d)  Haar  Transform 
Crispened,  k = 1. 


Figure  4-4.  FLIR  imagery  — ship  (close-in). 
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a)  Original.  b)  Local  Area  Histogram 

Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness 
and  Gain  Control, 
statistical  gain  16/4. 


d)  Haar  Transform 
Crispened,  k = l. 


Figure  4-5. 


FLIR  imagery  — tank  (close-in). 


I 
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a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness 
and  Gain  Control, 
statistical  gain  16/4. 

Figure  4-6.  FUR  imagery  — power  plant  (short  range). 


d)  Ilaar  Transform 
Crispened,  k - 1 . 
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a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness  d)  Haar  Transform 

and  Gain  Control,  Crispened,  k = 1. 

statistical  gain  16/4. 

Figure  4-7.  FLIR  imagery  — tank  (short  range). 
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I a)  Original, 
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b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 
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c)  Local  Area  brightness  d)  Ilaar  Transform 

and  Gain  Control,  Crispened,  k - 1. 

statistical  gain  16'4, 

Figure  4-8.  Radar  imagery  — mountains. 
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a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness 
and  Gain  Control, 
statistical  gain  16/4. 


d)  Haar  Transform 
Crispened,  k = 1. 


Figure  4-0.  Radar  imagery—  road. 


a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness 
and  Gain  Control, 
statistical  gain  16/4. 


d)  Haar  Transform 
Crispened,  k = 1. 


t Figure  4-10.  Radar  imagery  - fields. 
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a)  Original. 


b)  Local  Area  Histogram 
Equalized,  4x4 
center  box. 


c)  Local  Area  Brightness 
and  Gain  Control, 
statistical  gain  16/4, 


d)  Haar  Transform 
Crispened,  k - 1 . 


Figure  4-1!.  Radar  imagery  — river. 
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• Orientation 

• Target  detection 

• Target  recognition 

• Target  identification 

• Target  designation 

6.  Evaluation  rating  scale  — The  imagery  was  evaluated  using  the 
following  scale: 

7 Excellent 

6 Very  good 

5 Good 

4 Acceptable 

3 F air 

2 Poor 

1 Not  usable 

The  original  sensor  image  was  defined  as  acceptable  (4)  for  all 
scenes.  The  enhanced  images  were  rated  relative  to  the  original. 

Results 

Each  subject  evaluated  the  eleven  tactical  scenes  relative  to  the  spe- 
cific operational  tasks  pertinent  to  the  scene.  The  average  rating  scores  for 
the  three  enhancement  techniques  are  presented  in  Tables  4-1  (LAHE), 

4-2  (LABGC),  and  4-3  (HTC).  Comparative  data  for  the  three  sensors  are 
depicted  in  Figures  4-12  (TV),  4-13  (FLIR),  and  4-14  (radar). 

Viewer  rating  scores  were  strongly  influenced  by  sensor,  scene,  and 
task  factors  in  addition  to  the  enhancement  technique  being  used.  The  impact 
of  these  factors  is  reflected  in  the  following  data  summaries. 

• Sensor  - The  average  ratings  by  sensor  are  as  follows: 


TV 

FLIR 

RADAR 

LAHE 

4.  6 

4.  35 

4.  0 

LABGC 

4.9 

4,4 

3.  7 

HTC 

4.2 

3.8 

3.  55 

AVE 

4.  57 

4.  20 

3.  75 
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Figure  4-13.  Average  ratings  — FLIR. 
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For  each  enhancement  technique,  the  TV  images  were  evaluated 
as  being  most  improved  over  the  original,  FLIR  images  were 
judged  as  better  with  LAHE  and  LABGC  enhancement,  but  less 
acceptable  with  HTC  enhancement.  Enhancement  had  virtually 
no  effect  on  radar  images  (LAHE  and  LABGC)  except  for  the 
HTC  technique  which  reduced  image  acceptability. 

Scene  - Ratings  varied  widely  for  the  different  scenes.  The  three 
highest  lated  scenes  by  enhancement  technique  were  as  follows: 

LAHE  2(5.4)  1(4.9)  4 & 5 (4. 5) 

LABGC  2(5.  I)  5(5.0)  1 & 3 (4.8) 

HTC  1(4.5)  4(4.2)  2 (4.1) 

In  general,  close-in  and  medium  range  images  (1,  2,  4,  and  5) 
benefitted  most  from  enhancement. 

Operator  task  - In  specific  cases  such  as  target  identification  (TV) 
high  ratings  were  achieved  for  all  enhancement  techniques. 

Highest  overall  averages  for  individual  tasks  were  recorded  for 
the  target  identification  and  target  designation  tasks.  As  shown 
by  the  task  averages  (Tables  4-1,  4-2,  and  4-3),  the  highest 
ratings  were  as  follows: 


LAHE 

TGT  ID 

TGT  DESIG 

4.  7 

4.  6 

LABGC 

TGT  ID 

TGT  DESIG 

4.  6 

4.  3 

HTC 

TGT 

TGT  ID 

DESIG 
and  VEH 
CONT 

3.  9 3.  8 

These  data  indicate  that  operator  tasks  requiring  use  of  high 
image  detail  benefit  most  from  image  enhancement. 

Enhancement  technique  - In  summary,  the  data  from  this 
evaluation  indicate  that  for  detailed  interpretation  tasks,  using 
optical  sensors  at  short  and  medium  ranges,  image  enhancement 
improves  the  viewability  and  utility  of  the  sensor  image.  The 
results  are  more  pronounced  in  favor  of  the  LAHE  and  LABGC 
techniques,  with  the  impact  of  the  HTC  technique  being  minimal. 


4.  2 HARDWARE  IMPLEMENTATION  EVALUATION 

This  section  describes  the  hardware  implementation  of  each  of  the 
image  enhancement  techniques  (LAHE,  LABGC,  HTC).  These  mechanizations 
are  constrained  to  run  at  real-time  TV  rates.  A functional  block  diagram  of 
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each  technique  is  given  along  with  an  estimate  of  the  number  of  integrated 
circuit*. 

This  evaluation  concluded  that  the  three  enhancement  technique* 
investigated  in  this  study  can  be  mechanized  with  state-of-the-art  hardware. 
The  LABGC  and  HTC  are  both  much  easier  to  mechanise  than  the  LAHE 
technique. 

To  estimate  the  size,  power  and  weight  of  each  mechanization  based 
on  the  number  of  ICs,  the  following  figures  were  used. 

Size  - Height  = 8" 

Width  = 6" 

Length  = (ICs)  x 0.02  "+2" 

Power  = (ICs)  x 0.  2 watts 
Weight  = (ICs)  x 0.  015  pounds 

Table  4-4  summarizes  the  data  collected  in  this  evaluation. 

Histogram  Equalization 

The  two  dimensional  sliding  window  histogram  equalization  mechani- 
zation is  shown  in  Figure  4-15.  It  consists  of  digital  memory  for  the  storage 
of  a partial  frame  of  the  video  plus  storage  for  the  histogram  itself  and 
special  logic  to  generate  the  histogram  and  control  the  truncation  process. 
Table  4-5  reflects  the  complexity  of  this  process  by  defining  the  number  of 
integrated  circuits  for  the  various  functions. 


TABLE  4-4.  SUMMARY  OF  ENHANCEMENT  TECHNIQUES 
AS  TO  HARDWARE  COMPLEXITY. 


Enhancement 

Technique 

Hardware  Complexity 

ICs 

Size 

(Inches) 

Weight 

(Pounds) 

Power 

(Watts) 

LABGC 

330 

6x8  x 8.  6 

5 

66 

LAHE 

770 

6 x 8 x 17.4 

11.  5 

154 

HTC 

384 

6 x 8 x 9.  7 

5.8 

77 
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Figure  4-15.  Histogram  Equalization  Block  Diagram, 


TABLE  4-5.  HISTOGRAM  EQUALIZATION 
MECHANIZATION  COMPLEXITY 


Function 


Input  Buffer 
Code  Conversion 
Histogram  Storage 
Output  Buffer 
Timing  and  Control 


No.  ICs 


Total 


Implementation  of  the  local  area  histogram  equalization  algorithm 
requires  32  lines  of  video  storage  to  generate  the  32  line  by  32  element 
histogram.  This  32  x 32  histogram  is  sub-divided  into  8 smaller  histo- 
grams that  are  4 elements  by  32  lines  called  minihistograms.  To  histo- 
gram equalize  at  TV  rates,  multiplexing  of  these  histograms  is  necessary. 
This  minimizes  the  memory  . equirements.  Each  larger  histogram  of 
32  elements  by  32  lines  is  used  to  equalize  the  center  4 element  by  4 line 
array.  The  histogram  is  then  shifted  horizontally  by  4 elements  to  convert 
the  next  inner  4x4  array.  Therefore  each  successive  32  x 32  histogram 
differs  by  one  4 x 32  minihistogram.  Forming  the  first  histogram  requires 
the  summing  of  8 minihistograms.  Each  additional  histogram  is  generated 
by  adding  one  and  subtracting  another  minihistogram. 

Multiplexing  of  the  transform  memories  is  required  to  allow  time 
to  update  the  transformation  data.  Since  each  minihistogram  is  used  in 
multiple  histograms  the  transform  memories  must  be  updated  simultaneously. 
This  requires  the  video  conversion  process  to  occur  in  parallel  necessitating 
a output  buffer  to  convert  the  timing  to  standard  T.  V. 

Local  Area  Brightness  and  Gain  Control 


The  mechanization  of  this  process  as  shown  in  Figure  4-16  consists 
of  memory  and  processing  circuitry  to  derive  the  optimum  brightness  (bias) 
and  gain  (contrast)  of  the  video  based  on  its  local  characteristics.  The 
mechanization  complexity  estimate  is  summarized  in  Table  4-6. 
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Figure  4-16.  LABGC  Block  Diagram, 


TABLE  4-6.  LOCAL  AREA  BRIGHTNESS  AND  GAIN  CONTROL 
MECHANIZATION  COMPLEXITY 


Function 

No.  ICs 

Buffer 

30 

Ave  Bright 

140 

Sync  Gen 

10 

Timing  and  Control 

30 

Variance 

120 

Total 

330 

The  process  is  implemented  by  first  storing  eight  lines  of  video  in  a 
recirculating  shift  register  memory.  Vertical  shifting  of  the  statistical 
window  is  accomplished  by  updating  this  buffer  one  line  at  a time.  Horizon, 
tal  shifting  is  done  by  adding  the  difference  between  the  newest  and  the 
oldest  video  elements  to  the  statistics  of  the  window.  The  average  brightness 
is  then  summed  over  eight  lines  and  subtracted  from  the  center  element  of 
the  window  area. 

The  video  with  average  brightness  removed  is  digitally  multiplied 
by  a gain  term.  A multiplexer  is  used  to  select  either  a constant  gain,  or 
a gain  which  is  inversely  proportional  to  the  variance  of  the  statistics.  The 
variance  is  calculated  by  the  same  process  as  the  mean  except  the  video  is 
squared  before  processing  and  then  subtracted  from  the  mean.  The  variance 
calculation  is  implemented  digitally  because  analog  techniques  for  computing 
variance  for  a sliding  window  of  statistics  are  very  complicated. 

Haar  Transform 

The  Haar  transform  mechanization  as  shown  in  Figure  4-17,  consists 
of  buffer  memory  for  the  storage  of  input  digitized  video  and  the  intermediate 
and  final  transformed  results.  The  actual  transform  and  inverse  transforms 
are  performed  using  high  speed  arithmetic  logic  circuits  capable  of  per- 
forming  the  required  shifts,  adds  and  subtracts.  Table  4-7  reflects  the 
complexity  of  this  approach  in  terms  of  the  number  of  integrated  circuit 
components  required  to  mechanize  the  various  functions. 
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Figure  4-17.  Block  Diagram  Haar  Transform, 


TABLE  4-7.  HAAR  MECHANIZATION  COMPLEXITY 


Function 

No.  ICs 

Sync  Gen 

10 

Input  Buffer 

12 

Haar  Xform 

137 

Xform  Buffer 

48 

Inverse  Xform 

137 

Timing  and  Control 

40 

384 

As  a line  of  video  is  received  in  the  processor,  it  is  stored  in  the 
input  buffer.  The  Haar  Transform  is  performed  and  the  transformed  data 
stored  in  the  Haar  memory.  During  the  horizontal  blanking  time  the  desired 
shifting  function  is  applied  to  the  transformed  video.  As  the  second  line  of 
video  is  received,  stored,  and  transformed,  the  inverse  transformation  is 
performed  on  the  first  line  and  subtracted  from  the  unprocessed  stored 
line  in  memory.  The  third  line  is  stored  in  the  first  buffer  as  the  first 
line  is  being  sent  to  the  display. 

The  Haar  transformation  is  implemented  by  taking  the  sum  and 
difference  of  alternate  pixels  of  video.  Difference  terms  are  final  terms 
while  sum  terms  are  processed  again.  Each  successive  process  generates 
2^”^  terms  where  i = 0 to  7.  The  inverse  transformation  is  identical  except 
the  process  reverses  the  order  of  the  sums  and  differences. 


5.0  CONCLUSION 


Three  image  enhancement  techniques  were  investigated,  optimized, 
and  evaluated  in  this  study: 

1.  Local  Area  Brightness  and  Gain  Control  (LABGC), 

2.  Local  Area  Histogram  Equalication  (LAHE),  and 

3.  Haar  Transform  Crispening  (HTC). 

The  LABGC  technique  manipulated  brightness  and  gain  within  local 
image  areas.  It  was  found  to  work  best  when  the  gain  was  a function  of  the 
local  standard  de/iation  and  the  brightness  a function  of  both  the  local  mean 
and  standard  deviation.  This  combination  of  gain  and  brightness  control 
produced  good  enhancements  in  most  cases. 

Applying  histogram  equalization  to  local  image  areas,  as  in  the  LAHE 
technique,  showed  tremendous  improvement  over  the  full  frame  equalization 
approach.  One  refinement  was  made  to  the  LAHE  technique.  This  involved 
shifting  or  biasing  the  mean  of  the  equalized  imagery  so  that  it  more  closely 
approximated  the  mean  of  the  unenhanced  imagery.  This  logic  was  added 
because  in  certain  cases,  histogram  equalization  biases  the  enhanced  imagery 
toward  the  dark  end  of  the  intensity  scale. 

Computer  simulations  showed  that  image  crispening  could  be  achieved 
using  the  Haar  transform.  Analysis  was  used  to  derive  a crispening  operator 
composed  entirely  of  binary  shifts.  This  resulted  in  a technique  that  produced 
edge  sharpening  by  a combination  of  simple  addition,  subtraction,  and  binary 
shifts. 
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Evaluation 

A human  factors  evaluation  of  the  enhancement  techniques  was 
performed.  This  subjective  analysis  produced  the  following  conclusions: 

1.  Viewer  acceptance  and  utility  of  enhanced  imagery  is  dependent 
upon  the  sensor  being  used*  the  scene*  and  the  operator  task. 

2.  Enhanced  imagery  is  most  useful  in  detailed  interpretation  tasks. 

3.  Enhancement  techniques  produced  the  best  results  with  TV  type 
imagery  and  good  results  with  FUR  scenes. 

4.  Enhancement  techniques  are  of  unknown  value  when  applied  to 
synthetic  aperture  radar  imagery. 

This  evaluation  strongly  indicates  that  image  enhancement  techniques 
can  be  utilized  to  improve  viewer  interpretation  performance  of  critical 
tactical  tasks  which  involve  overall  target  area  orientation*  target  detection* 
identification  and  designation.  These  results  are  more  pronounced  in  favor 
of  the  histogram  and  LABGC  techniques*  with  the  impact  of  the  Haar 
technique  being  minimal. 

The  hardware  implementation  evaluation  indicates  that  the  LABGC 
and  HTC  techniques  are  the  easiest  to  mechanize.  The  HTC  is  estimated 
to  be  slightly  more  difficult  to  mechanize  than  the  LABGC. 

In  conclusion,  this  study  showed  that  the  LABGC  and  LAHE  techniques 
can  improve  operator  performance  of  sensor  display  systems.  This  result 
is  very  pronounced  for  electro-optical  sensors.  The  Haar  transform  crispen- 
ing  technique  showed  little  overall  improvement  in  operator  performance. 
Radar  imagery  was  affected  little  by  enhancement  and  in  some  cases  actually 
degraded  the  image.  The  LABGC  technique  is  the  easiest  to  mechanize  in 
hardware  while  the  LAHE  is  the  most  difficult.  Table  5-1  summarizes  these 
results. 

FOR  THE  FUTURE 

Future  work  should  be  done  to  optimize  the  window  sizes  of  both  the 
LABGC  and  LAHE  techniques.  Surprisingly*  the  LABGC  technique  did 
slightly  better  than  the  LAHE.  It  was  expected  that  the  LAHE  technique 
would  do  better  since  it  non-linearly  reassigns  gray  scale  levels  based  on 
image  statistics.  The  LABGC,  on  the  other  hand,  is  a linear  technique  also 
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TABLE  5-1.  SUMMARY  OF  ENHANCEMENT  TECHNIQUES  AS  TO 
IMAGE  IMPROVEMENT  AND  HARDWARE  COMPLEXITY. 


Enhancement 

Technique 

Average 

Image 

Improvement 
4. 0 = Original 

Hardware  Complexity 

ICs 

Size 

(Inches) 

Weight 

(Pounds) 

Power 

(Watts) 

LABGC 

4.4 

330 

6x8x8. 6 

5 

66 

LAHE 

4.3 

770 

6x8x17. 4 

11.5 

154 

HTC 

3.85 

384 

6x8x9. 7 

5.8 

77 

using  image  statistics.  However,  the  extra  capability  of  the  LAHE  technique 
to  adaptively  enhance  contrast  would  seem  to  have  been  an  advantage.  This 
expectation  would  most  likely  be  realized  with  further  optimization  of  window 
size. 

The  effects  of  the  enhancements  of  the  radar  imagery  on  operator 
performance  were  inconclusive.  A possible  explanation  is  that  these  images 
were  logarithms  of  the  original  scenes.  This  logarithmic  process  is  common 
in  ground  mapping  radar  displays  due  to  the  large  dynamic  range.  Applying 
the  enhancement  techniques  to  the  original,  or  linear,  radar  imagery  and 
evaluation  by  experienced  radar  photointerpreters  would  likely  yield  improved 
results. 

The  Haar  transform  crispening  (HTC)  technique  didn't  perform  as  well 
as  expected  overall.  This  may  have  been  the  result  of  using  fairly  crisp 
images  to  start  with.  An  investigation  of  the  HTC  technique  using  images  that 
possess  real-world  blur  due  to  misfocused  optics  or  MTF  roll  off  in  the  sensor 
electronics  would  yield  new  results.  Also,  the  derivation  of  a Haar  domain 
crispening  operator  matched  to  a specific  resolution  loss  problem,  would 
most  likely  lead  to  marked  improvement  in  operator  performance.  This  may 
in  turn  permit  development  of  systems  in  which  lower  cost  and  lower  quality 
components  can  be  used  with  no  decrease  in  performance. 

Finally,  an  evaluation  of  a selected  image  enhancement  technique 
should  be  conducted  with  real-time  man-in-the-loop  simulation,  or  flight 
tests.  This  would  yield  quantitative  data  on  the  cost  effectiveness  of  real- 
time image  enhancement  techniques  for  sensor  display  systems. 
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